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ABSTRACT: A core goal for most learning analytic projects is to move from small-scale research 
towards broader institutional implementation, but this introduces a new set of challenges 
because institutions are stable systems, resistant to change. To avoid failure and maximize 
success, implementation of learning analytics at scale requires explicit and careful consideration 
of the entire TEL technology complex: the different groups of people involved, the educational 
beliefs and practices of those groups, the technologies they use, and the specific environments 
within which they operate. It is crucial not only to provide analytics and their associated tools, 
but also to begin with a clear strategic vision, assess institutional culture critically, identify 
potential barriers to adoption, develop approaches that can overcome these, and put in place 
appropriate forms of support, training, and community building. In this paper, we offer tools and 
case studies that will support educational institutions in deploying learning analytics at scale with 
the goal of achieving specified learning and teaching objectives. The ROMA Framework offers a 
step-by-step approach to the institutional implementation of learning analytics and this approach 
is grounded here by case studies of practice from the UK and Australia 
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1 INTRODUCTION 


Learning analytics are concerned with the measurement, collection, analysis, and reporting of data 
about learners and their contexts, for purposes of understanding and optimizing learning and the 
environments in which it occurs (SoLAR, 2011). The intention is to develop models, algorithms, and 
processes that can be widely used. Transferability is a key factor here; analytic and predictive models 
need to be reliable and valid at a scale beyond the individual course or cohort. It is also important that 
analytics have demonstrable benefits for learners and educators and do not distract or mislead them. 

There are currently few reports in the learning analytics literature of deployment at scale. In England 
and Australia, standardized testing of schoolchildren has been employed for decades, through the 
English SAT tests and the Australian NAPLAN tests (ACARA, 2013; Kirkup, et al., 2005). These tests are 
aligned with stated government aims, make use of agreed proxies for learning, provide clear and 
standardized visualizations of analytics, and drive behaviour at every level of the education system. The 
data generated by these tests are collected, analyzed, and reported with the intention of optimizing 
learning and the environments in which it occurs. Despite the scale of this deployment, media reports 
suggest that many educators, learners, and parents have not been convinced that these programmes 
have demonstrable benefits for learners and educators (Bantick, 2012; Harrison, 2010). 

One of the best-known examples in the learning analytics literature of implementation at scale has 
taken place at Purdue University in the United States. By 2012, the university had applied its Course 
Signals analytics tool to over 100 courses, providing formative grade feedback to over 23,000 students 
(Arnold & Pistilli, 2012). Development of the Course Signals tool had its roots in a study carried out at 
Purdue in 2005 (Campbell, 2007). In spite of significant investment over a nine-year timeframe, Course 
Signals has not yet been deployed across the entire university. It has, however, been deployed widely, 
and is now available as a commercial product from Ellucian. Similar systems are sold by other vendors, 
and are used by many universities and colleges, but none of those has published robust, peer-reviewed 
research on its effects. The work of Arnold and Pistilli (2012) and others has highlighted the efficacy and 
positive impact of Course Signals on student retention and has sparked much interest and investment in 
learning analytics. However, bloggers in the field have questioned the research methodology and 
interpretation of results employed in this work. They note, for example, that the analysis did not control 
for the number of courses taken (Caulfield, 2012, 2013; Essa, 2013). A more robust analysis is still 
required in order to confirm any positive effect on retention. 

Educational institutions aiming to incorporate learning analytics face several challenges. At the most 
basic level, they need to understand what learning analytics are, how they could benefit the institution 
and what drawbacks they have. There is often confusion between learning analytics, used to support 
learners and teachers, and academic analytics, used by administrators to support comparisons between 
departments or institutions. Griffiths (2013) identifies how the use of academic analytics could disrupt 
the balance within a university, "analytics techniques have the potential to disturb the balance between 
educational managers and the practice of teaching professionals, by extending the ability of the former 
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to gather and process information about the latter." 

At the time of writing, only a few universities have made use of learning analytics at scale, and there are 
therefore no detailed implementation accounts available. Another problem is that, although numerous 
small-scale projects have reported successful outcomes (see the Proceedings of the Learning Analytics 
and Knowledge (LAK) conference series for examples), many of these analytics are heavily dependent on 
contextual local factors, and there is as yet no strong evidence of the overall effectiveness of learning 
analytics deployment at scale. Organizational leaders must therefore require not only the necessary 
understanding of analytics, but also the vision to see how such small-scale projects and pilots might be 
successfully scaled to improve teaching and learning across an institution. 

In this paper, we discuss a range of barriers to institutional change and present case studies, planning 
frameworks, and models that offer institutional teams practical guidance in the implementation of 
learning analytics. Section 2 surveys the factors associated with resistance to analytics approaches in 
higher education. Section 3 proposes that systems theory offers useful insights into complex educational 
systems and introduces the TEL Innovation Process, which identifies components of educational system 
that must be addressed when carrying out educational innovation projects. Section 4 introduces the 
RAPID Outcome Mapping Approach (ROMA) as a structured framework for the implementation of 
learning analytics that considers these different components. Section 5 moves from theoretical 
frameworks to practical implementation, and includes case studies showing how learning analytics are 
being embedded successfully across the University of Technology Sydney and The Open University. 
Finally, Section 6 unites these approaches at different levels. 


2 BARRIERS TO ANALYTICS IMPLEMENTATION 

Beyond a "lack of good examples," what other barriers to the implementation of institutional learning 
analytics exist? In the UK, The Open University has been engaged in learning analytics-like work since its 
foundation, and has also been identifying barriers to the implementation of that research for more than 
three decades (McIntosh, 1979). More recently, researchers seeking to inform institutional decision 
making through provision of analytic data and insights have identified additional factors that may hinder 
adoption. 

In 1979, McIntosh reported that "those of us in the Survey Research Department [at The Open 
University] continue to be dissatisfied at our ability to have an impact on many major problem areas," 
making this statement long before the development of learning analytics. McIntosh was engaged in the 
related area of educational evaluation — delineating, obtaining, and providing information that would 
be useful in judging decision alternatives. She identified reasons why competent research findings were 
never put into practice, including unwillingness of academics to accept and act on methods or findings 
from outside their own research area, individual preferences for qualitative or quantitative approaches, 
a tendency to base decisions on anecdote rather than on research, the different forms of discourse used 
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by researchers and decision makers, an unfamiliarity with statistical methods on the part of decision 
makers, and a tendency by researchers to hedge their conclusions. 


McIntosh's recommendations focused on the need for researchers and decision makers to work 
together: “Researchers should get clients politically, emotionally, and financially committed to the 
outcome of the research. They are then more likely to take notice of its results" (1979). While the focus 
of her article was on university decision makers as clients in the project of educational evaluation, her 
findings and suggestions have direct relevance for the field of learning analytics, where the clients, or 
stakeholders, include learners, educators, and administrators. 


In 2012, Macfadyen and Dawson reported on the non-implementation of a study relating to an 
institution's use of learning analytics and its learning management system (LMS). They found that the 
institutional planning process was dominated by technical concerns and, because of this, "made little 
use of the intelligence revealed by the analytics process." After the current-state analysis had been 
completed and noted by the institution's standing committee on learning technologies, minutes and 
reports show that no references to or discussions of the findings were made in subsequent meetings. 

These authors suggested that powerful analytic findings and conclusions were set aside because the 
strategic planning process for which these analytics were commissioned gave no attention to the 
institutional culture of higher education. It also had little awareness of the degree of resistance to 
change, and failed to embrace standard approaches for motivating change within an organization that 
might have fruitfully employed analytic output as part of a process of evidence-based decision making. 
They suggest that "greater attention is needed to the accessibility and presentation of analytics 
processes and findings so that learning analytics discoveries also have the capacity to surprise and 
compel, and thus motivate behavioural change" (Macfadyen & Dawson, 2012). 

What impact do all these barriers have? Globally, education lags behind other sectors in harnessing the 
power of analytics (Manyika et al., 2011). The first survey of analytics implementation in US higher 
education in 2005 found that, of 380 institutions, 70 percent were at "Stage 1" of a five-stage 
implementation process: "Extraction and reporting of transaction-level data" (Goldstein & Katz, 2005). 
Four years later, a study of 305 US institutions found that 58 percent were at Stage 1, while only 20 
percent reported progress to Stage 2: "Analysis and monitoring of operational performance" (Yanosky, 
2009). More recently, investigators have reported that while 70 percent of surveyed institutions agreed 
that analytics is a major priority for their school, the majority had yet to move beyond basic reporting 
(Bichsel, 2012; Norris & Baer, 2013). 

Clearly, learning analytics researchers face a significant challenge, since their primary focus is on issues 
such as the development and testing of algorithms and visualizations. When they develop analytics that 
can support learning and teaching, few analytics projects will have the capacity to undertake an 
ethnographic study of institutional culture or a review of recent thinking on change management. Few 
will have team members with experience of writing a research report that compels its audience to 
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action. Yet the learning analytics community needs to investigate these issues and to engage its 
audience, if it is to achieve its aim of optimizing learning and the environments in which it occurs. 

Lonn and colleagues (2013) undertook some initial investigation into the issues encountered and lessons 
learned when scaling up a learning analytics intervention. The focus of their reflection was the benefits 
and challenges of institutional partnership between a research team and a technology service group. 
Their study identified gaps between the two teams in areas such as usability, access, performance, and 
calculation. In each case, they identified possible solutions, although many of these solutions were 
specific to the context in which they were working. 

Overall, although they employ different language and describe different situations, McIntosh, Lonn, 
Macfadyen, and Dawson all identify some common problems. These relate to different expectations 
around communication between researchers and those responsible for implementation, different levels 
of engagement with the research, and different expectations about the role and purpose of educational 
research. These discrepancies are found in other areas of technology-enhanced learning (TEL) research, 
and it is increasingly clear that significant innovation in this area is not possible without taking into 
account the entire TEL Technology Complex (Scanlon et al., 2013). 


3 BARRIERS TO ANALYTICS IMPLEMENTATION 

Learning analytics theorists propose that learning analytics can and should permit optimization of the 
"learning system" (SoLAR, 2011). We argue in this paper that a systemic perspective is critical for 
successful implementation at scale of any educational innovation, including learning analytics. 

Educational institutions are superb examples of complex adaptive systems (CASs). That is to say, they 
exist as dynamic networks of interactions, made up of nested and clustered sets of similar "sub¬ 
systems." As adaptive systems they, and their component sub-systems, have the capacity to learn and 
change in response to conditions, and can display self-organizing behaviours and emergent properties. 
CASs evolve and have a history that is "co-responsible" for their current state — their present culture 
and structures are shaped by history and experience (Cilliers, 1998; Gupta & Anish, 2009; MacLennan, 
2007; Mitleton-Kelly, 2003). 

Like all systems, educational institutions are resilient in the face of perturbation, and exist far from 
equilibrium, requiring a constant input of energy to maintain their organizational structure and 
processes (see Capra, 1996). These properties of interconnectedness and interdependence mean not 
only are these establishments resistant to change, but also that change strategies aimed at only one or a 
few of their subsystems are unlikely to succeed. (For a more detailed discussion of policy development 
for complex educational systems, see Macfadyen et al. (2014), and references therein.) 

Identifying and describing the critical components of the CAS that is an educational institution is a 
challenge. The TEL Technology Complex model outlined by Scanlon and her colleagues (2013) offers an 
approach that highlights the many components of the "technology complex" of higher education: the 
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series of components that must all be addressed together to understand the whole (Fleck & Howells, 
2001). Key components of the complex besides pedagogy include stakeholders, communities, current 
practices, context, technical components, and business model (see Figure 1). When scaling up learning 
analytics, all these components need to be taken into account. 
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Figure 1: The TEL Technology Complex (Scanlon et al., 2013) 

The TEL model in Figure 1 also begins to identify relationships and connections between components of 
the system. The introduction of an innovation such as learning analytics requires changes to the 
practices of several communities at once in order to complete the TEL Innovation Process (Figure 2). 
Moreover, the TEL innovation process stresses that, for an innovation process to be successful, a vision 
and articulated strategy for educational change are required on the part of the institution, together with 
a commitment to persistent work towards this vision over time. Success also requires a willingness to 
engage in bricolage — a term that refers to working with the people and resources available in the 
context and linking them in ways that support work towards the vision. 
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Figure 2: The TEL Innovation Process (Scanlon et al., 2013) 

The TEL Innovation Process (Figure 2) makes it clear that piecemeal, simplistic, and non-systemic 
approaches to learning analytics implementation will struggle to gain traction across an institution. 
Analytics implementation requires a change to most practices across an educational institution. 
Educators need to be able to evaluate any implementation of analytics tools in order to use them 
effectively. Learners need to be convinced that analytics are reliable and will improve their learning 
without unduly intruding into their privacy. Support staff need to be trained to maintain the 
infrastructure and to add data to the system. Library staff need to be able to use the analytics to shape 
their practice and resources. University administrators need to be convinced that the implemented 
analytics provide a sound return on investment and demonstrably improve teaching and learning 
quality. IT staff need to put workflows into place so that raw data are collated, prepared for use, and 
made readily available to end users. In order to convince all these stakeholders to put in the sustained 
effort necessary to make use of learning analytics, a clear vision of the gains to be made is required at 
the outset and should be maintained throughout (Scanlon et al., 2013). 

What are needed are nuanced planning and implementation approaches, developed for complex 
systems, which explore and address the interconnected challenges of learning design, leadership, 
institutional culture, data access and security, data privacy and ethical dilemmas, technology 
infrastructure, and the existing gap in institutional analytics skills and capacity. Such approaches allow 
development of fluid strategies that can adapt to changing context, take advantage of policy windows, 
permit implementation of effective monitoring and learning systems, and keep abreast of their ever- 
changing dynamics (Young & Mendizabal, 2009). Moreover, comprehensive systemic approaches offer 
the potential to identify points of intervention (Corvalan, Kjellstrom, & Smith, 1999), with the goal of 
offering educational leaders and practitioners additional insight and tools in their project of improving 
the system with learning analytics. 
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4 THE RAPID OUTCOME MAPPING APPROACH (ROMA) 

The TEL Technology Complex model and the TEL Innovation Process models identify broad areas that 
must be addressed during the implementation of learning analytics at scale, but offer little practical 
guidance for the systematic development of strategy and policy for learning analytics implementation. 

Here, we have selected and adapted an existing schema, ROMA (Figure 3), that has been developed for 
complex contexts (Young & Mendizabal, 2009), as a model for guiding an iterative approach to planning 
the systemic institutional implementation of learning analytics. Like Macfadyen & Dawson (2012), the 
developers of the ROMA model note that "Facts alone — no matter how authoritative — may not be 
enough" to maximize the impact of research on policy and practice (p. 1). 

Originally developed to support policy and strategy processes in the field of international development, 
the seven-step ROMA model is focused on evidence-based policy change. It is designed to be used 
iteratively, allowing refinement and adaptation of policy goals and the resulting strategic plans over time 
and as contexts change. As a systemic approach, it is designed to support the development of a holistic 
understanding of context, including external and internal influences, political and cultural context, 
evidence, and the links between "all of the other actors and mechanisms that affect how the evidence 
gets into the policy process" (Young & Mendizabal, 2009). 
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4. Develop 
engagement 
strategy 


Figure 3. The RAPID Outcome Mapping Approach (ROMA) 
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Developed by the Research and Policy in Development Programme (RAPID) of the UK Overseas Development 
Institute, core steps in the ROMA model are represented in the central cycle of the figure. Boxed lists for each step 
refer to additional tools and materials that may also be useful. These are available at http://www.odi.org.uk/rapid 

4.1 Adapting the Model for Learning Analytics 

The seven-step ROMA model has been lightly adapted for the learning analytics context: 

1. Define a clear set of overarching policy objectives 

2. Map the context 

3. Identify the key stakeholders 

4. Identify learning analytics purposes 

5. Develop a strategy 

6. Analyze capacity; develop human resources 

7. Develop a monitoring and learning system (evaluation) 

Step 1: Define a Clear Set of Overarching Policy Objectives 

What are our objectives for learning analytics? What changes do we seek to achieve? This approach to 
systemic policy and strategy development begins with collaborative development of a vision through the 
formulation of policy objectives. As statements of intent or vision, policies can be implemented as 
procedures, strategies, or protocols. Young & Mendizabal (2009) propose that development teams 
consider the following kinds of change that might be required: 

• Discursive change (e.g. changing how information is communicated and shared); 

• Procedural change (e.g. changing how something is done: how decisions are made, how 
learners are supported); 

• Content change (e.g. changing written policy with regard to evidence-based support of 
learners); 

• Attitudinal change (e.g. changing how key stakeholders perceive the project); and 

• Behavioural change (e.g. making sustainable changes in the way student success is achieved or 
supported). 

Step 2: Map the Context 

Section 2 identified barriers to the adoption of evidence-based approaches such as learning analytics. 
The process of mapping context seeks to uncover barriers that exist in the local context. It examines: 

• The "political context": — The people, institutions, and processes that may help or hinder 
change. It is important to consider whether there is political interest in change, and how key 
decision makers may perceive the problem(s) that learning analytics could address and/or the 
proposed solutions; 

• The evidence: — Whether evidence exists that could convince others of the need for change, 
and how this can best be presented; 

• Links" — The people and processes that affect whether this evidence can be effectively 
introduced into the policy process. "Are there key organisations and individuals with access to 
policy makers, are there existing networks to use?" (Young & Mendizabal, 2009, p. 3). 
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The context-mapping step is critical in any change project, because it allows organizations to identify the 
key factors specific to their context that may influence the implementation process, and therefore has 
the potential to illuminate points for intervention. 

Step 3: Identify the Key Stakeholders 

Who will benefit from the use of learning analytics? Long-term institutional goals may include the 
implementation of analytics strategies, tools, and processes to serve multiple stakeholders. However, 
financial and logistical realities require the development of a plan that prioritizes selected stakeholders 
and purposes. Identifying stakeholders in the system should inform strategic planning and the design of 
approaches to involve, inform, support and train key players. 

Step 4: Identify Learning Analytics Purposes 

Learning analytics may fulfill a range of purposes, including: 

• Learner awareness 

• Monitoring and tracking 

• Reflection and research 

• Evaluation and planning 

• Reporting and communication (adapted from Kay, 2013). 

Stakeholders and purposes are tightly connected, and not all stakeholders have the same needs or 
goals. Given the extensive range of learning analytics possibilities, it is critical to prioritize stakeholders 
and goals realistically in order to meet institutional goals, and to account for limitations of time, 
resources, and/or budget. 

Step 5: Develop a Strategy 

The strategic plan must identify what needs to be done in order to meet the desired outcomes. Strategic 
planning for learning analytics necessitates work across multiple domains. This may include: 

• Planning strategies to intervene at points identified in the context-mapping work undertaken in 
Step 2 

• Considering data needs, access and availability, based on Step 4 decisions 

• Anticipating ethical dilemmas; establishing data policy and governance processes 

• Reviewing and planning technology infrastructure to support data generation, extraction, 
warehousing and integration 

• Planning a learning design strategy to maximize production of meaningful data 

• Developing strategies to engage leaders, promote buy-in and change educational culture 
Through integrating strategic planning across domains, a preliminary action plan and timelines can be 
developed and these can be reviewed and modified as needed. 

Step 6: Analyze Capacity, Develop Human Resources 

Does the institution have the capacity to implement the planned strategy? Many commentators 
(Manyika et al., 2011; Siemens, Dawson, & Lynch, 2013) have pointed to the skills gap that may hamper 
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implementation of learning analytics. To achieve its goals in this area, an institution is likely to require 
the following skill-sets: engagement/management, data science, qualitative analysis, project evaluation, 
database development, learning technologies administration, IT support, front-end / interface 
development, analytics design and development, learning analytics reporting, data visualization, data 
governance development and management, data policy development and implementation, institutional 
reporting / business intelligence. Some individuals will possess more than one group of skills; in other 
cases, several individuals will be required to undertake the scope of work envisioned in Steps 1-5. The 
diversity in the range of skills and capacities required for broad scale deployment of analytics re¬ 
enforces that implementation of learning analytics requires a process of bricolage. 


Step 7: Develop a Monitoring and Learning System (Evaluation) 

As Young and Mendizabal (2009) and others note, evaluation processes are important, "not only to track 
progress, make any necessary adjustments and assess the effectiveness of the approach, but also to 
learn lessons for the future" (p. 4). 

The ROMA model, as adapted for institutional implementation of learning analytics, guides a cyclical and 
iterative process of mapping, planning, and review. Part of the monitoring step calls for first principles to 
be revisited. Are the original policy objectives and vision still accurate and relevant in the light of the 
assessment of context, purposes, and capacity? 

Longer-term evaluation also calls for post-implementation evaluation strategies to determine whether 
learning analytics are achieving the desired changes. Outcomes-based evaluation approaches (see, for 
example, UNESCO, 2011) are especially useful for projects focused on educational transformation. These 
approaches clearly emphasize the difference between "project activities" and "project outcomes." 
When our focus is on improving learning, the critical results we need to monitor and measure are the 
results that reflect positive educational change. 

Sections 3 and 4 have focused on generic ways of approaching the implementation of learning analytics 
across an institution. Section 5 examines how the process plays out in practice. The University of 
Technology Sydney and The Open University in the UK are based on opposite sides of the globe. One is a 
face-to-face institution; the other is designed for distance education. Despite their differences, each is 
committed to building an effective, institution-wide approach to learning analytics. We describe these 
approaches to three case studies. In two of these, we set out the project in terms of the ROMA 
framework throughout. In Case Study 1A, we do this in detail; in Case Study 3, we give another example, 
giving an outline of an extensive project in terms of the ROMA framework. In Case Study IB, we begin by 
outlining the learning analytics work and then showing how a pre-existing project such as this can be 
understood in terms of the framework. 
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5 IMPLEMENTATION ACROSS THE INSTITUTION 
5.1 Case Study 1A: The Open University, UK: Data Wranglers 

The Open University (OU) is a distance education establishment with over 200,000 students and 10,000 
academic and non-academic staff, which has been engaged in educational evaluation and learning 
analytics since its foundation more than 40 years ago. It has recently undertaken two significant 
programmes of activity, both explicitly rooted in learning analytics. The first, Case Study 1A, is the 
deployment of Data Wranglers, and the second, Case Study IB, is a far-reaching Strategic Investment 
Project in Learning Analytics, led by senior management. 

Data Wranglers are academic staff who explore a range of data related to student learning and present 
their findings to staff in the university faculties with actionable recommendations. Their role as human 
data interpreters who help to close the feedback loop is set out at length elsewhere (Clow, 2014). 

The university recognized that increasing volumes of educational data (student feedback, activity in the 
Moodle VLE/LMS, information on the mode of course delivery, and aggregated demographics and 
outcomes) were available but were not being used effectively. There was no integrated, systematic view 
being developed to inform and enhance teaching and learning practice. Pilot work in 2010 and 2011 led 
to the launch of the Data Wrangling project in 2012. The activity was not originally developed with 
explicit reference to the ROMA framework (Section 4), but it is useful to analyze it in those terms. 

Step 1: Define a Clear Set of Overarching Policy Objectives 

The objectives of the project were located firmly within the existing policy and planning framework of 
the university. The original objectives were (1) to develop a group of staff with expertise in the individual 
faculty contexts, (2) to set up a system for collating, synthesising, and reporting on the available data, (3) 
to produce reports at regular intervals, and (4) to build strong relationships with the faculties. In Young 
and Mendizabal's (2009) terms, the primary kinds of change sought were discursive changes to how the 
data were communicated and shared, and procedural changes to how decisions were made in 
curriculum development and student support. 

Step 2: Map the Context 

The project leaders were familiar with the complex organizational context. Data wrangling was located 
in an existing unit, the Institute of Educational Technology (IET), with a mission that included analyzing 
and influencing teaching and learning practice. IET already had responsibility for curation and 
presentation of some of the data concerned. Many Data Wranglers had pre-existing connections to 
specific faculties, and the unit had good access to senior management. There was already significant 
interest and engagement in learning analytics and data at a senior level, and the relationship between 
the Data Wrangling project and the development of the broader analytics strategy was identified as a 
crucial one at an early stage. 
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Step 3: Identify the Key Stakeholders 

Given the scope of the data, all members of the university were considered stakeholders. However, the 
project focused particularly on delivering insight into curriculum-development and quality-enhancement 
processes. The unit leading this work was also engaged in rolling out learning design across the faculties; 
so learning design work could be grounded in evidence from learning analytics. Many key contacts were 
the same in both cases. The unit was also engaged in initiatives that targeted other aspects of the 
learning process. The key stakeholders were therefore the senior managers in each faculty with 
responsibility for learning and teaching and/or curriculum development. Curriculum development at the 
OU takes place through module teams, and these were also identified as stakeholders. Other key 
stakeholders included senior management and those responsible for data gathering and curation. 


Step 4: Identify Learning Analytics Purposes 

The project focused on curriculum development and quality enhancement. In addition to the drivers 
discussed above, this focus was influenced by considerations related to the data. Student feedback and 
final outcomes data (completion and pass rates) are released twice a year. The curriculum development 
and quality enhancement processes at the OU follow a similar cycle. This provides two points each year 
when the Data Wrangling project could integrate available data with data from Moodle, the university's 
virtual learning environment. Moodle systems do not yet support real-time monitoring, and so real-time 
changes were beyond the scope of the project. 

Step 5: Develop a Strategy 

Extensive consultation and feedback led to the development of an implementation strategy. Early pilot 
work helped inform the shape of the Data Wrangling project. The host unit had a robust project 
management system, and the project developed documentation that included both a plan for 
implementation, and dates for review. 


Step 6: Analyze Capacity, Develop Human Resources 

Capacity analysis was an explicit part of the project planning. For the Data Wranglers themselves, 
developing a full understanding of the faculty teaching and learning context was among the original 
objectives. Training in advanced use of Microsoft Excel was arranged for the Wranglers. Considerable 
time was spent in exploring and understanding the data, including liaison with those responsible for its 
collection and curation. New technical tools were deployed and developed (including Tableau 
Workbooks and SAS Stored Procedures presenting data from the Data Warehouse via the Intranet), 
which necessitated further staff development. On the "client" side, one aim of the project was to 
develop an understanding and appreciation of what the data could show, as well as an awareness of 
how to access it without the mediation of a Data Wrangler. 


This was an iterative process. It was not straightforward for the Data Wranglers to understand some of 
the data, and how it could be interpreted. Some of the issues they encountered were resolved clearly, 
some were determined to be hard-to-fix data quality issues, and others remain as puzzles. 
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Step 7: Develop a Monitoring and Learning System 

Feedback from stakeholders was built into the process of delivering the reports. In addition, an explicit 
evaluation exercise in July 2013 gathered feedback from key stakeholders, and this informed further 
development. 

The project was extremely time intensive, both in terms of staff time and in terms of the delay between 
the end of a course and the Data Wrangler's report being available. Reports also proved to be very 
different in terms of coverage and quality. To some degree, this was a positive feature, as each Wrangler 
negotiated and developed a shared understanding with client stakeholders. Some faculties were 
confident in reading a range of data visualizations; others were interested in qualitative analysis that 
would help them to understand not only what was happening, but also why it was happening. Some 
large faculties wanted to see their data broken down in various ways, while smaller faculties were 
interested in seeing the whole picture. 

Much has been learned, and a further review was undertaken in summer 2014, with the aim of 
streamlining the process. The project has enabled a better understanding of what data are of use to the 
curriculum development and quality enhancement processes. Tools have been developed that can 
deliver data reports quickly and reliably, to high standards, with minimal manual intervention. This will 
reduce the time demands on the Data Wranglers, leaving them free to explore the data further in order 
to answer pressing questions from faculty clients. 

This section has outlined some of the ways in which the Data Wrangler project has achieved a degree of 
success in fostering adoption at scale. A key to this success was that it was well integrated with existing 
systems, processes, and networks from the outset. Stakeholder engagement at all levels was critical. The 
project required significant allocation of staff resources, including resources from the faculty "client" 
stakeholders. 

This account of the Data Wrangler project has been aligned throughout with the ROMA framework, in 
order to provide a worked example of the process. We now turn to a related project, the institution's 
Strategic Analytics Investment Programme. In this case, we first describe the programme as a whole and 
then show how its development can be related to the ROMA framework. The intention is to show that 
wherever a university is in the process of learning analytics deployment, the ROMA framework can be 
used to understand and develop that process. 


5.2 Case Study IB: The OU Strategic Analytics Investment Programme 

With extensive datasets and a desire to improve learner outcomes, the OU has embarked on an eight- 
strand programme of work to promote the use of learning analytics for learner benefit. The programme 
is sponsored by the Pro Vice-Chancellor Learning and Teaching and takes into account the needs of 
multiple stakeholders, which include university administrators, students, and educators. 
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The Strategic Analytics Investment Programme was launched in 2012 and brought disparate groups 
across the University together with a single vision: to use and apply information strategically (through 
specified indicators) to retain students and enable them to progress and achieve their study goals. The 
intention was to undertake work to achieve this at two levels: 

• Macro-level work aggregates information about the student learning experience at an 
institutional level in order to inform strategic priorities that will improve student retention 
and progression; 

• Micro-level work makes use of analytics to drive short, medium, and long-term 
interventions. 

In order to achieve the programme's vision, three key areas are dependent on each other and underpin 
the work. These are (1) analysis and creation for insight, (2) availability of data and, (3) processes that 
have an impact on student success (Figure 4, below). 

The vision and related action are informed by an understanding of data in action, data on action, and 
data for action. Multiple stakeholders draw upon data in action through a live portal that enables them 
to understand learner behaviour and make adjustments and interventions that will have an immediate 
positive impact. Data on action is a more reflective process that takes place after an adjustment or 
intervention. Data for action takes advantage of predictive modelling and innovation in order to isolate 
particular variables and make changes based on a variety of analysis tools. 



Figure 4: Underpinnings of the OU Strategic Analytics Model (Tynan & Buckingham Shum, 2013). 
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This evidence-based approach to student success, encapsulated in Figure 5 (below), allows for flexibility 
and constant evaluation of all actions. Key success outcomes and lead indicators are defined through the 
four stages of student recruitment, retention, progression, and completion. Measures are drawn 
together from "learning and teaching" and "student support activities," and are available to a range of 
stakeholders. Stakeholders make use of integrated analytics to inform interventions designed to 
improve outcomes. 

These interventions are evaluated and then become the evidence base for factors that drive student 
success. For example, as part of the university's quality assurance process, a "module pass rates model" 
is used to compare actual module pass rates with those expected based on a statistical analysis of the 
previous achievement of students over the preceding five years. Use of the model has given the 
university an improved understanding of the characteristics and behaviours of students who are more 
likely to struggle with their studies. The module pass rates model ensures that key stakeholders can 
implement appropriate support interventions for both short- and long-term benefits. 


Another strategic action was the launch, in 2014, of a reconfiguration of the OU's student support 
approach, along with a new data tool that enables subject-specific student support teams to trigger 
interventions with students based on analysis of their progress, using data related to their demographic 
characteristics, assignment submission, and online activity. 
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Figure 5: Evaluation cycle of the OU Strategic Analytics Model (Tynan & Buckingham Shum, 2013). 
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The programme of work that supports the move towards student success has seven strands. These 
deliver the tools practitioners need, within key business cycles, to increase student persistence through 
the development and implementation of targeted and evidence-based interventions. 

This approach has created a community of stakeholders who, led by a senior executive, depend on each 
other for best effect. Data are managed holistically from one database to ensure the best possible 
quality, and reporting assumptions are agreed across the programme of work. This has ensured a joined- 
up approach to how the university goes about the deployment of analytics. 

The seven strands of the programme are: 

1. Intervention and Evaluation 

• The university uses analysis of current student performance to identify priority areas for 
action, both in terms of changes to curriculum and learning design, and in terms of 
interventions with the students most at risk of not progressing with their studies. 

• A common methodology is being used to evaluate the relative value of interventions 
through measuring the resulting student behaviours and performance that inform future 
improvements to the student experience. 

2. Data Usability 

• Simple data visualizations are being built around key performance measures. These will be 
available in near real time to key stakeholders in order to monitor student performance. 

• A new analytics self-service portal triangulates different data sources, enabling academics 
and student support staff to identify patterns and some of the factors, such as subject or 
geographical area, that influence success in their context. 

3. Ethics Framework 

• A Learning Analytics Ethics policy details what data is being collected and its ethical use to 
improve educational processes and support individual students. 

4. Predictive Modelling 

• Machine-learning-based predictive models are currently live in several subject areas within 
the university and provide a weekly prediction of each student's likelihood of submitting 
their next assignment based on analysis of key factors, including online activity. 

5. Learning Experience Data 

• In future, the university will collect feedback during modules, rather than relying on surveys 
carried out at the end of each study module. This will enable academics and student support 
staff to react more quickly to any issues faced by students. 
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• By systematically connecting data on the learning design of study modules with student 
activity data, the university will examine how those learning designs impact on the success 
of its varied student base. 

6. Professional Development 

• A professional community of practice focused on the retention and progression of year one 
students uses an evidence hub to share best practice across faculty boundaries. 

7. "Small Data" Student Tools 

• "Small data" connects people with timely, meaningful insights (derived from big data and/or 
"local" sources), organized and packaged — often visually — to be accessible, 
understandable, and actionable for everyday tasks. Tools that put actionable information, 
based on analytics, in the hands of students are being developed in order to help students 
keep track of their own progress and make the right study choices as they move through 
their degree. 

Strategic Analytics at the OU and the ROMA Framework 

This high-level account of the OU approach provides an example of how institution-wide analytics are 
implemented in practice. It also offers an opportunity to show how pre-existing work can be understood 
in terms of the steps of the ROMA framework set out at the beginning of Section 4.1 

At Step 1, the overarching policy objectives for the OU include discursive changes to the communication 
of data and analytics within the institution, to procedural changes in how learners are supported, and to 
behavioural changes associated with sustainable change in learner support. Figure 5 provides some 
examples of the ways in which context was mapped (Step 2). The programme defines key stakeholders 
broadly as university administrators, students, and educators (Step 3). It also broke these stakeholder 
groups down further, as the discussion of programme "strands" implies. 

The purpose of the learning analytics (Step 4) is clearly defined in terms of using and applying 
information strategically in order to retain students and support them to achieve their study goals. This 
is to be achieved by a carefully thought-through strategy (Step 5) applied at both macro and micro 
levels, and structured around data in action, data on action, and data for action. 

Step 6, analyzing capacity and developing human resources, is institution-specific and so is not covered 
in the summary above, but has included such elements as recruitment, capacity building, and 
developing an ethical framework for the use of learning analytics. Finally, monitoring (Step 7) is carried 
out through a process of constant evaluation, with attention to specific success outcomes and leading 
indicators. 

In this case, an analysis of the programme in terms of the ROMA framework shows that each step was 
included. In other cases, the framework could be used to identify steps that have been omitted and thus 
to recommend future actions that could strengthen existing work. 
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5.3 Case Study 2: The University of Technology, Sydney, Australia 

The University of Technology, Sydney (UTS) is an inner-city university with a vision to become a world¬ 
leading university of technology. In keeping with that vision, it embarked on a project in 2011 to become 
a "data intensive university" (DIU). This case study reports on the strategy used and its progress to date. 
As with OU Case Study 1A, described above, the UTS strategy was not initially developed with an explicit 
reference to the ROMA framework, but analysis of its current progress and successes show that the UTS 
approach maps well onto this systemic planning framework. 

The UTS project was launched in the belief that access to data can enrich all aspects of the university 
and provide a springboard for creation and innovation. Recognizing the importance of data analytics to 
contemporary university practice, UTS first developed an operational definition that outlines what it 
means to be a DIU: 

A university where staff and students understand data and, regardless of its volume and 
diversity, can use and reuse it, store and curate it, apply and develop the analytical tools to 
interpret it. 

Flowing from this definition, the DIU strategy was designed with the goal of making better use of data to 
enable students, staff, alumni, and industry partners to explore and thrive; to understand their 
environment, solve issues and challenges; to lead their fields; and to provide opportunities to develop 
knowledge. 

Step 1: Define a Clear Set of Overarching Policy Objectives 

The UTS project is guided by a broad analytics strategy, with objectives encompassing all aspects of the 
university's work: teaching and learning, research, and administration. 

In the case of teaching and learning, UTS aims to use learning analytics to improve student learning and 
to improve the student experience of university. It aims to ensure that all stakeholders have the capacity 
to understand and interpret contemporary data-rich environments. 

In research, the programme objectives include providing an environment that allows researchers to 
access and manipulate data more easily and effectively, and that also enables them to think and act 
differently when designing their research methodologies and practices. 

In administration, the major objective is to identify opportunities to obtain, generate, visualize, and 
communicate data and analyses that can improve decision-making capability and improve core business 
outcomes. 

At the university level, the strategy focuses on the importance of mining existing institutional data to 
identify areas that can provide direct evidence or assistance to staff and students. For example, data and 
analytics can be provided for staff to facilitate the design of a set of intervention strategies that will 
address students at risk of withdrawing from a course of study prior to completion. 
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Step 2: Map the Context 

The project was initiated, and has been led by, a member of the Senior Executive of the university, the 
Deputy Vice-Chancellor and Vice-President (Teaching and Learning). She initially obtained pilot funding 
and, after successful completion of a number of pilot projects, has secured ongoing funding. The project 
was initially run from her office, but the ongoing funding for the project has enabled the establishment 
of a Connected Intelligence Centre. An internationally renowned learning analytics professor has been 
recruited as its inaugural director. Critical to the success of the initial pilot projects was the existence of 
an Advanced Analytics Institute 1 with internationally regarded researchers in big data, data sciences, 
and analytics sciences. 

Step 3: Identify the Key Stakeholders 

In order to gain the level of ongoing funding needed to ensure the longevity of the initiative, it was 
critical to achieve a broad level of support across the whole university, particularly from the senior 
executive, deans, and directors of relevant units. The idea of becoming a "data intensive university" was 
first raised at a senior staff retreat at the beginning of 2011 and support was given to a scoping project. 

In the latter part of 2011, approximately 190 UTS staff (150 present and 40 online) attended a one-day 
"Data Intensive University Forum," thus beginning a university-wide conversation. Although there was 
almost universal buy-in to the ideas of the project, a major point of contention was the naming of the 
initiative. Although the phrase "data intensive" is well established in some fields of science, it was 
thought to have the potential to alienate academics in other fields of study and create barriers to 
acceptance for many people. For this reason, the name "data intensive university project" was replaced 
by "connected intelligence project." 

A working party was established, chaired by the Deputy Vice-Chancellor (Teaching and Learning), with 
the Deputy Vice-Chancellors for (Research) and (Corporate Services) as deputy chairs. A senior member 
of the library staff was seconded to the project as the senior manager. Each faculty was represented on 
the working party, as was each of the administrative areas with relevant expertise. 

Achieving stakeholder buy-in and ongoing participation in the project have been critical to its success. 

Step 4: Identify Learning Analytics Purposes 
Learning analytics is being used or will be used to: 

• Provide information that can be used to decrease student attrition; 

• Provide a more detailed understanding of factors affecting low pass rates in subjects with 
very high failure rates over time, referred to as "killer subjects"; 

• Provide students with more information about their own study and engagement patterns 
through a personalized dashboard; 

1 http://www.uts.edu.au/research-and-teaching/our-research/advanced-analvtics-institute 
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• Enable a more fine-grained understanding of the influences of a range of possible 
interventions on pass rates and completions: e.g., what is the influence of the peer-assisted 
study scheme on pass rates and retention over time; 

• Provide valuable input to learning futures projects encompassing personalization of learning 
through adaptation and intervention. 


Step 5: Develop a Strategy 

From the beginning of the project, elements of strategy have been designed to: 


• Give attention to institutional culture, —ensuring engagement and buy-in from key stakeholders 
through good communication and governance; 

• Invest in pilot projects of significant concern to the university and reporting of outcomes; 

• Invest in infrastructure:—tools, applications, services; 

• Invest in expertise:—recruitment of critical staff; 

• Provide leadership and engage institutional leaders. 

Step 6: Analyze Capacity, Develop Human Resources 

As UTS becomes a more data-intensive institution, one of the most critical factors in its success will be 
ensuring that analytics stakeholders have the capacity to understand data, to make judgements about 
its meaning, and thus to engage in evidence-based decision making. There is no point in making such a 
significant investment if students and staff are not sufficiently numerate and equipped to make use of 
the analyses that analytics projects produce. 

For this reason, a subject has been developed and trialled to develop students' "ability to engage with 
complex, extended arguments underpinned by numerical data as a key to participation as informed 
citizens in issues of significance to our culture and society." The subject has been trialled twice with staff 
and from next semester will become available as an elective before being made compulsory thereafter. 
The intention is to continue this practice to increase the numeracy levels of staff. 

Step 7: Develop a Monitoring and Learning System (Evaluation) 

Much has already been learned from early pilot projects. For example, the Outreach Program makes 
telephone contact with as many commencing undergraduate students as possible. Early results have 
consistently shown a significant decrease in attrition in the group of students contacted. Without 
funding to contact every commencing student, analytic techniques have been used to identify those 
students considered most at risk and they now receive priority for telephone contact. 

In parallel, the "killer subject" project identified several areas for attention by the course coordinator. 
These issues have now been addressed and the failure rates have significantly decreased. 

To date, UTS has been engaged in a variety of learning analytics projects to assess scale and impact, 
under the auspices of the broader DIU project. While this project remains a multidimensional work in 
progress, the degree of institutional buy-in and funding committed suggests that the systematic 
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strategic planning approach employed is contributing to project success and the integration of analytics 
into institutional culture. 


6 CONCLUSION 

This paper began by considering the goals of learning analytics and the barriers and challenges that 
make implementation of learning analytics at scale a complex task. The literature shows that some of 
these barriers have existed in educational institutions for decades, particularly those relating to 
communication challenges between researchers and those responsible for implementation, to different 
levels of engagement with research, and to different expectations about the role and purpose of 
educational research. Perhaps even more pressing for institutions currently interested in learning 
analytics implementation at scale is the reality that few good examples of such implementation exist. 
This means there is little guidance available to help institutions navigate the complexities of such an 
enormous change process. 

The Rapid Outcome Mapping Approach (ROMA) provides a framework for structuring plans for large- 
scale implementation and adoption processes. The seven-step framework for learning analytics 
adoption detailed here takes users from initial policy objective to final evaluation. The move from 
theoretical frameworks to operational practice is a difficult step. To demonstrate the value of this 
approach and to show how it can be employed in practice, we have grounded the ROMA approach in 
case studies from the UK and Australia. These case studies illustrate how the framework can be applied 
on a systematic basis, or used to support a process already well underway. Our hope is that, in future, 
these tools and case studies will give institutions, departments, and faculties the confidence to 
implement learning analytics at scale in order to achieve their specified learning and teaching objectives. 
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